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Abstract — This paper presents a blind detection and com- 
pensation technique for camera lens geometric distortions. 
The lens distortion introduces higher-order correlations in 
the frequency domain and in turn it can be detected using 
higher-order spectral analysis tools without assuming any 
specific calibration target. The existing blind lens distortion 
removal method only considered a single-coefficient radial 
distortion model. In this paper, two coefficients are consid- 
ered to model approximately the geometric distortion. All 
the models considered have analytical closed-form inverse 
formulae. 

Key Words: Radial distortion, Geometric distortion, Lens 
distortion compensation, Higher order spectral analysis. 

I. Introduction 

Generally, lens detection and compensation is modelled 
as one step in camera calibration, where the camera cal- 
ibration is to estimate a set of parameters describing the 
camera's imaging process. Using a camera's pinhole model, 
the projection from the 3-D space to the image plane can 
be described by 
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where the matrix A^ntr denotes the camera's intrinsic ma- 
trix, (a, 7, /3, Up, Wo) are the camera's five intrinsic param- 
eters, with (a, (3) being two scalars in the two image axes, 
(uq, vq) the coordinates of the principal point (which is also 
assumed to be at the center of distortion in this paper), 
and 7 describing the skewness of the two image axes. In 
the imaging process described by ^ , the effect of lens dis- 
tortion has not been considered. 

In equation (0), {u, v) is not the actually observed image 
point since virtually all imaging devices introduce certain 
amount of nonlinear distortions. Among the nonlinear dis- 
tortions, radial distortion, which is performed along the 
radial direction from the center of distortion, is the most 
severe part P, 0- In the category of polynomial radial 
distortion modeling, the commonly used radial distortion 
function is governed by the following polynomial equation 
0, i, 0, 0, 0, 0: 

ra = r+Sr = r fir,k) = r {l+kir^+k2r^+hr'' + - ■ ■), (2) 

where k = [/ci, fc2, fcs, . . .] is a vector of distortion coefh- 
cients. r and can be defined cither in the camera frame 



as 0, m, 0, 



with 
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or on the image plane as 0, 



ruv = wo)2 + {v - Vq)"^, 

J'duv = V i'^d - UqY + {Vd - VqY, 



(3) 



(4) 



(5) 



where the subscript d denotes the distorted version of the 
corresponding ideal projections. The subscripts uv and xy 
denote the definitions on the image plane and the cam- 
era frame, respectively. The distortion models discussed 
in this paper are in the Undistorted-Distorted formulation 
ini, though similar idea can be applied to the Distorted- 
Undistorted formulation. 

Most of the existing camera calibration and lens distor- 
tion compensation techniques require either explicit cal- 
ibration target, whose 2-D or 3-D metric information is 
available or an environment rich of straight lines T". 
The above mentioned techniques are suitable for situations 
where the camera is available. For situations where di- 
rect access to the imaging device is not available, such as 
when down loading images from the web, blind lens removal 
technique has been exploited based on frequency domain 
criterion |10| . The fundamental analysis is based on the 
fact that lens distortion introduces higher-order correla- 
tions in the frequency domain, where the correlations can 
be detected via tools from higher-order spectral analysis 
(HOSA). However, it has been reported that the accuracy 
of blindly estimated lens distortion is by no means com- 
parable to those based on calibration targets. Due to this 
reason, this approach can be useful in areas where only 
qualitative results are required [117). 

The paper is organized as follows. Section starts with 
an introduction of the blind lens distortion removal tech- 
nique, which is based on the detection of higher-order cor- 
relations in the frequency domain. In Sec. IIIII we briefly 
describe two existing polynomial/rational radial distortion 
approximation functions, along with our recently developed 
simplified geometric distortion modeling method. Sec- 
tion ^3 presents detailed lens distortion compensation re- 
sults using images from both calibrated cameras and some 
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web images. The previously calibrated cameras act as a 
validation of the blind lens removal technique. Finally, sec- 
tion concludes the paper. 

II. Frequency Domain Blind Lens Detection 

In this section, higher-order spectral analysis is first re- 
viewed, which provides the fundamental criterion for the 
blind lens distortion removal technique ^Oli 03 - The ba- 
sic approach of the blind lens distortion removal exploits 
the fact that lens distortion introduces higher-order cor- 
relations in the frequency domain, which can be detected 
using HOSA tools. 

A. Bispectral Analysis 

Higher-order correlations introduced by nonlinearities 
can be estimated by higher-order spectra [TJ. For example, 
third-order correlations can be estimated by bispectrum, 
which is defined as 



B{loi,U2) ^ E{F{uji)F{uj2)F*iuji + UJ2)}, 



(6) 



where E{-} is the expected value operator and F{u!) is the 
Fourier transform of a stochastic one-dimensional signal in 
the form of 



F{u;)^ ^ f{k)e 



-jujk 



(7) 



Notice that the bispectrum of a real signal is complex- 
valued. Since the estimate of the above bispectrum has the 
undesired property that its variance at each bi-frequency 
{oji,uj2) is dependent of the bi-frequency, a normalized bis- 
pectrum, called the bicoherence, is exploited, which is de- 
fined to be [Tn | . [TT ] 



6^(cJl,LJ2) 



\B^{u;i,L02)\ 



E{\F{LUl)Fiuj2W}E{\F{LUl+UJ2m' 

The above bicoherence can be estimated as 



(8) 



b{uji,u;2) 



(9) 

which becomes a real- valued quantity. As a measure of the 
overall correlations, the following quantity is employed in 

COl 

N/2 N/2 . , 

u]i = -N/2uj2 = -N/2 ^ ' 

where TV is the dimension of the input one-dimensional 
signal. 

B. Blind Lens Removal Algorithm 

Consider a signal fd(x) that is a distorted version of f{x) 
according to 



with K controlling the amount of distortion. It has been 
shown in JHI that correlations introduced by the nonlin- 
earity is proportional to the distortion coefficient k, where 



the quantity is chosen as the measure of the correla- 
tions. Now, consider the inverse problem of recovering f{x) 
from fd{x). It is only when k, is properly estimated that 
the inverted f{x) contains a least amount of nonlinearities, 
in which case f{x) holds a minimum bicoherence. 

Based on the above discussions, an intuitive algorithm 
applied for the blind lens distortion removal is listed in the 
following |10| : 

1) Select a range of possible values for the distortion coef- 
ficients K. 

2) For each perform inverse undistortion to fd{x) yield- 
ing a provisional undistortion function (x) . 

3) Compute the bicoherence of f^ix). 

4) Select the k that minimizes all the calculated bicoher- 
ence of the undistorted signals. 

5) Remove the distortion using the distortion coefficient 
obtained from step 4). 

III. Lens Distortion Modeling 

The radial distortion model applied for the blind lens 
distortion removal in jlT)) is 



^duv — ^uv(l ~t- ^uv ^uv)' 

which is equivalent to 

(Ud - Uo) = (u - Uo)(l + kuv ?'uv)> 

{vd ~ vq) = {v - Vo){l + kuvrl^), 



(12) 



(13) 



when assuming the center of distortion is at the principal 
point. 

Besides the radial distortion modeling, we have recently 
proposed a simplified geometric lens distortion modelling 
method, where lens distortion on the image plane can be 
modelled by [T^l 



{ud - Uq) (li - Uq) /(j'uv, kuvl), 
{vd - Vq) = {v - Vo) /(ruv, kuv2), 



(14) 



where /{r^v, ^uvi,2) can be chosen to be any of the available 
distortion functions. To illustrate the simplified geometric 
distortion idea, in this paper, /{r^v, k^^i) and /{r^v, k„^2) 
are chosen to be the following rational functions [?]: 



/(^uvj kuvl) — 
firuv, fcuv2) 



1 



1 -|- ^uvl^uv 
1 "h ^uv2^uv 



(15) 



for its fewer number of distortion coefficients and the prop- 
erty of having analytical geometric undistortion formulae. 
From equations (|14(l and H15(l . we have 



(11) (ud - Uo)^(l + fcuvlfuv)^ + {vd - Wo)^(l + fcuv2?'uv)^ = ''uv, 



with r 



uv — ' uv 



(16) 

The above equation is a quadratic function 



in Tuv, thus having analytical inverse formula. 
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IV. Experimental Results 

In this section, we first verify that the bhnd lens removal 
technique, which is based on the detection of higher-order 
correlations in the frequency domain, can be used for the 
detection and compensation for lens distortion. This veri- 
fication is via the comparison of the calibration coefficients 
of the blind removal technique with those calibrated by a 
planar-target based calibration method 0]. Though, the 
calibration results by the blind removal technique are by 
no means comparable to those based on calibration target, 
the results shown in Sec. IIV-^ does manifest a reasonable 
accuracy, at least for applications where only qualitative 
performance is required. 

The existing blind lens distortion removal method only 
considered a single-coefficient radial distortion model, as 
described in In Sec. HV^ we show that cameras, 

which are more accurately modelled by different distortion 
coefficients along the two image axes, can also be detected 
using higher-order correlations. As an example, the simpli- 
fied geometric distortion modeling with the distortion func- 
tion H15II is applied. Using single-coefficient to describe the 
distortion along each image axis, totally two coefficients 
are used in (|15(l . One reason for choosing the function (|15|l 
is in its fewer number of distortion coefficients that reduces 
the whole optimization duration. Another advantage lies 
in that equation H15() has analytical geometric inverse for- 
mula, which further advances the optimization speed. 

Finally, lens distortion compensation of several web im- 
ages are performed using the blind removal technique. 

A. Verification of Blind Lens Compensation Using Cali- 
brated Cameras 

Comparisons between the distortion coefficients cali- 
brated by the blind removal technique with those ob- 
tained by the target-based calibration method have been 
performed in ^Uj using distortion function (|12|) . Simi- 
lar comparison is given here via two calibrated cameras, 
the desktop |T3 and the ODIS camera using the dis- 
tortion function (|12|l . along with the function l|15l) with 
fcuvi = fcuv2- We think that this double verification is 
needed since lens nonlinearity detection using higher-order 
spectral analysis is a recent development. 

Original images of the desktop and the ODIS cameras 
are shown in Fig. ^ where the plotted dots in the center of 
each square are used for judging the correspondence with 
the world reference points for the target-based calibration. 
Using the single-coefficient radial distortion model in (|15|l 
(with kuvi = fcuv2), the blindly compensated images of the 
two cameras are shown in Fig. [3 An image interpolation 
operator is applied during the lens distortion compensa- 
tion, since the observed original images are shrinked due 
to the negative distortion coefficients. The undistorted im- 
ages are only plotted in gray-level to illustrate the lens 
compensation results^. Comparing Fig. ^ with Fig. El it 

^The image interpolation operator currently applied is an average 
operator around each un-visited pixel in the compensated image. The 
resultant undistorted images might have noise and blur due to this 
simple operator. 



can be observed that lens distortion is reduced significantly, 
though not completely and perfectly. 
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Fig. 1 

Two Sample images of the model plane with the extracted 

CORNERS (indicated BY CROSS) FOR THE DESKTOP AND ODIS 
CAMERAS. 



Using the planar calibration target observed by the cam- 
eras as shown in Fig.Q] the desktop and ODIS cameras have 
been calibrated in 12 , [?] using the planar-based camera 
calibration technique described in jl]. However, the cali- 
brated camera parameters in |12| . [?] are in the normalized 
camera frame, while the blind removal technique deals with 
distortions in the image plane directly. A transformation 
between the lens distortion coefficients in the camera frame 
and those in the image plane is thus needed. 

A rough transformation is illustrated in the following 
using the obtained intrinsic parameters from the planar- 
target based calibration technique. From equation JQ), we 
have 



Assuming that 



7«0, 



a : 



/3, 



(17) 



for a coarse approximation and using a single-coefficient 
radial distortion model H12(l . we have 



rdu 



I xy 



(18) 



The relationship between k^v and fcxy can be determined 
straightforward as 



(19) 



In this paper, blind lens distortion compensation is im- 
plemented via Matlab using higher-order spectral analysis 
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toolbox following the procedures listed in Sec. III-BI How- 
ever, instead of using equation as the objective func- 
tion to minimize, the maximum value of bicoherence is used 
in our implementation as the measurement criterion for 
nonlincarity, which is^ 



J 



1 



N1N2 



, UJ2&I 



N 



(20) 

for an input image of dimension A^i x N2. 

Comparison of the distortion coefficients obtained from 
the blind removal and the target-based 0] calibration tech- 
niques is shown in Table. HI using the functions H12|l and 
l(T3|l with fcuvi = fcuv2 ■ It can be observed from Table 
that, despite the deviation of the blindly calibrated results 
from those based on calibration targets, there is a consis- 
tency about the trend qualitatively. Notice that the dis- 
tortion coefficients under the "Target" column in Table 
are obtained via approximations, where of course more pre- 
cise transformations can be achieved without applying the 
assumptions in H17() . However, since currently the blind 
removal method considers only the lens distortion (no cali- 
bration of the center of distortion), precise comparison can 
not be achieved even with precise calculation from the side 
of the target-based algorithm. Generally, the comparison 
can only be performed "quantitatively" , though the blind 
removal technique does provide another quantitative crite- 
rion for evaluating the calibration accuracy. 

^The reason to use the criterion in l2Ui is more experimental. In 
our simulati ons, distortion coefficients obtained via the average sum 
criterion in 1101 deviates from the previously calibrated coefficients 
signi fican tly. However, when using the maximum bicoherence crite- 
rion I2UI . close and reasonable calibration results can be obtained for 
both the desktop and the ODIS cameras. 



TABLE I 

Comparison of Lens Distortion Coefficients of the Blind 
Removal Technique and The Target-Based Algorithm 



Eqn. 


Camera 


Blind Technique (IQ-*^) 


Target 

(10-6) 


Values 


Mean 




Desktop 


-[3.5,4.5,3.5,4.5, 1.5] 


-3.5 


-3.73 


ODIS 


-[3.5,2.5,2.5,3.5,3.5] 


-3.1 


-4.13 


US* 


Desktop 


[5,6,5,5, 1] 


4.4 


4.27 


ODIS 


[5,3,4,5,4] 


4.2 


4.75 



*kuvl = kuy2 for modeling the radial distortion. 

One issue in the implementation is how to select the 
searching range for an image. While image normalization 
method is commonly applied, in our simulation, the search- 
ing range is determined based on the image's dimension and 
the observed judgement of radial or pincushion distortions 
in a non-normalized way. This is the reason why in Ta- 
ble n all the distortion coefficients are very small values. 
IVIore specifically, consider the radial distortion function 
(|12|l with the maximum possible distortion at the image 
boundary. Let r = rm ax and rg = pr^^^, where r^ax is 
defined to be rmax — a/mq -I- f g on the image plane and the 
subscript uv is dropped for simplicity. We have 

*uv = (21) 

max 

For our desktop and ODIS images, the dimension of the 
images is 320 x 240 in pixel. Further, the distortion ex- 
perienced by these two cameras is a barrel distortion with 
rd < r. Focusing on the barrel distortion by considering 



pe [0, 



, 1], we have 



, [-6.25x10 

fcuV - I 



-6 



when /9 = I , 
when p = 1. 



(22) 



The initial searching range for the distortion coefficients 
when using the radial distortion function H12|l is chosen 
to be within [—4.5 x 10"^ ^ 3.5 x 10^^] with a step size 
10^^. Similarly, for the radial distortion function l|15|) with 
^uvi — fcuv2, we have 



i-1 



(23) 



and 



(24) 



8.3 x 10-^ when p = |, 
0, when p = 1- 

The initial searching range when using function 115|) is [0 ^ 
9 x 10"^]. 

Relative values of J as defined in H20|l of the five ODIS 
images using the distortion functions H12|l and (|15|l with 
kuvi — kuv2 are shown in Figs. 13 and ^ The relative J val- 
ues equal to their corresponding values minus the minimum 
value in this group. 

B. Blind Detection and Compensation of Lens Geometric 
Nonlincarity 

Besides the radial distortion modelling, a simplified ge- 
ometric distortion modelling method has been developed 



5 



ODIS MODEL 1 




-1 I < < < < < < < < 1 

-5-4-3-2-101234 

Y in* 

Distortion Coefficient 
Fig. 3 

Relative J values of the ODIS images using function 1121 . 
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Fig. 4 

Relative J values of the ODIS images using function 11.51 . 



in A straightforward question asked is whether the 

higher-order correlation detection in the frequency domain 
can help for the detection of geometric distortions, instead 
of just the radial one. The above problem is pursued in 
the next, where we would like to first describe our conclu- 
sion. That is, the blind lens removal technique helps to 
detect the possible geometric distortion of a camera. By 
allowing the two distortion coefficients along the two im- 
age axes searched separately in two regions, the blindly 
calibrated lens distortion coefficients manifest noticeable 
difference for cameras that have been reported to be more 
accurately modelled by a geometric distortion modelling 
method. 

Again using the two calibrated desktop and the ODIS 
cameras, the blindly calibrated distortion coefhcients fcuvi 
and kuv2 using the rational distortion function l|15|l are 



shown in Table |nl where the difference between the two 
distortion coefficients is significant for the ODIS camera, 
which has been studied in 12_ to be better modelled by a 
geometric distortion model than a radial one. It is also ob- 
served that this difference between distortion coefficients 
along the two image axes is exaggerated using the blind 
removal technique. 



TABLE II 

Blind Detection of Geoivietric Distortion 





Desktop (10-^) 

[fcuvl J ^uv2] 


ODIS (10-6) 

[^uvl 1 fcuv2] 




[6.16,5.28] 


[2.64,6.16] 


Distortion 


[2.64,4.40] 


[4.40,6.16] 


Coefficients 


[5.28,6.16] 


[2.64,4.40] 




[2.64,3.52] 


[5.28,6.16] 




[5.28,3.52] 


[3.52,6.16] 


Mean 


[4.40,4.57] 


[3.70,5.81] 



In our implementation, geometric undistortion is imple- 
mented using equation l|15|l for each slice passing through 
the image center, which is chosen to be 12 degree apart 
for each image^. After {u,v) are derived from {ud,Vd) and 

(^uvl , ^uv2 

), where the (fcuvi, A:uv2) are determined through 
the searching procedures, r^v is calculate from (u, v) and 
the image center (which is assumed to be the center of 
distortion in the context of blind lens distortion removal 
technique). Nonlinearity detection using bicoherence is 
performed on this one-dimensional signal r^^, which is ba- 
sically the same procedures used in the blind removal of 
radial distortion. 

Due to the available knowledge of the distortion coeffi- 
cients for the two sets of images using the radial distortion 
modeling with function (|15|l for kuvi = kuv2, when per- 
forming the detection for possible geometric distortions, 
the searching ranges are chosen to be around the distor- 
tion coefficients already determined in the radial case. 

V. Concluding Remarks and Discussions 

Higher-order correlation based technique is a promis- 
ing method to detect lens nonlinearities in the absence of 
the camera. Besides the commonly used single-coefficient 
polynomial radial distortion model, blind detection of the 
geometric distortion is addressed in this paper. Using 
the quantitative measurement criterion defined in the fre- 
quency domain, difference between the two sets of distor- 
tion coefficients along the two image axes can be used to 
detect cameras that are better modelled by a geometric 
distortion modelling method qualitatively. 
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